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Abstract — In the future, remote images sent over communication 
lines will be reproduced in virtual reality(VR). This form of virtual 
telecommunications, which will allow observers to engage in an 
activity as though it were real, is the focus of considerable attention. 
The system will offer the experience of being in a place without having 
to physically go there. 

Taken a step further, real and unreal objects will be placed in a 
single space to create an extremely realistic environment. Here, 
imaginary and other life forms as well as people and animals in remote 
locations will gather via telecommunication lines that create a 
common environment where life forms can work and interact together. 
Words, gestures, diagrams and other forms of communication will be 
used freely in performing work. 

Actual construction of a system based on this new concept will not 
only provide people with experiences that would have been impossible 
in the past, but will also inspire new applications in which people will 
function in environments where it would have been difficult if not 
impossible for diem to function until now. 

This paper describes Tele Hyper Virtuality concept, its definition, 
applications, the key technologies to accomplish it and future 
prospects. 


Introduction 

In the future, remote images sent over information super highways 
will be reproduced in virtual reality(VR). This form of virtual 
telecommunications, which will allow observers to engage in an 
activity as though it were real, is the focus of considerable attention. 
The system will offer the experience of being in a place without having 
to physically go there. 

Taken a step further, real and unreal objects will be placed in a 
single space to create an extremely realistic environment called Hyper 
World. Here, imaginary and other life forms as well as people and 
animals in remote locations will gather via super highways, to a 
common environment called The Coaction Environment, where life 
forms can work and interact together. Words, gestures, diagrams and 
other forms of communication will be used freely in perfo rming work. 

Actual construction of a system based on this new concept will not 
only provide people with experiences that would have been impossible 
in the past, but will also inspire new applications in which people will 
function in environments where it would have been difficult if not 
impossible for them to function until now. 


This paper describes the concept, the technologies accomplishing 
it and the future prospects. 

Concept of Tele HyperVirtuality 

Inhabitants, such as people and animals in remote locations as well 
as imaginary and other life forms, will be able to coact; that is, they 
will be able to work and interact together, in a Hyper World where 
real, unreal and other worlds are fully integrated. 

Hyper World 

Hyper World is an advanced form of reality where real-world, 
computer graphic and other images are systematically integrated. 
Here, real-world images shot by camera and recognized by Computer 
Vision(CV) are realistically reproduced in Virtual reality(VR). These 
images may then be sent from remote location via super highways. 

Coaction environments 

Inhabitants, such as people, animals, and imaginary and other life 
forms, will be able to work and interact together using words, gestures 
and other forms of communication in the Hyper World environment 
This interaction is referred to as coaction. 

Coaction not only allows people in remote locations to work and 
play together as though they were in the same room, but it also allows 
people to interact with imaginary life forms. 

(1) Definition of a coaction environment 

This highly realistic environment provides interrelated objects 
with a common site, that is, a workplace or an activity area. The 
environment offers a means through which activities such as desi gnin g 
buildings, sharing activities and playing catch can be performed while 
communicating through words and gestures. The manipulation of 
physical bodies not only requires that objects conform to the laws of 
physics, such as moving like they are supposed to move, and changing 
shape when they collide, but that various life form activities take 
place, such as plants wilting and blooming according to their exposure 
to sunlight 

New environments are created by the interaction of coaction 
environments. In other words, multiple independent coaction 
environments interacting to form an integrated coaction environment 
exchange knowledge for integration. Sometimes these environments 
return to their original state after they stop interacting. 

Dynamic changes such as those just described are a feature of coaction 
environments. 
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(2) Advanced operation 

The basic function of an advanced operation is to enable activities 
such as automobile design or a recreational event using words, 
gestures, diagrams, voice and other forms of communication. 

(3) Coaction control 

This provides control functions and a site for interactive work and 
activities. The control functions include common area control, as well 
as integration, separation and other common area activities 
corresponding to the interaction of the inhabitants. 

Definition 

Technologies for creating highly realistic environments 
S: Highly realistic environment 

This is defined as follows. 

S - <S E , Sca, Scg. Scv> 

Sca : Nature, buildings and other objects shot with a camera 

Scg: Objects created through computer graphics 

Scv: Nature and other objects recognized and reproduced by 

Computer Vision 

Se: Real objects 

Definition of inhabitants 

I: People, animals, and imaginary and other life forms found in S 

This is defined as follows. 

I “ <lE, ICA> ICG> ICV> 

Ie: Real people and animals 

Ica: People and animals shot with a camera 

Icg-‘ Imaginary life forms created using computer graphics 

lev: People or animals recognized and reproduced by Computer 

Vision 



Coaction environments(CE) 

A coaction environment(CE) is a group of individual coaction 
environments(CEi) in a realistic space (S) where inhabitants(I) are 
coacting. 

CE- {CEi} 

A CEi is an individual environment where inhabitants(I) are 
coacting. 

Individual CEi dynamically integrate and separate repeatedly 
through CEi interaction. Coaction environments are shown in Fig. 1. 

A, B and C are coaction environments. 

If we assume that the ball in C rolls into B, the human figure in B 
picks up the ball and returns it to the human figure in C, then 
environments B and C are integrated to form coaction environment D. 
When the human figure in C accepts the ball and starts playing with the 
puppy, environments B and C are separated once again. 



Fig. 2 Medical Care 



Applications 


Implementation 


Tele Hyper Reality enables coaction activities in highly realistic 
environments connected by communication lines. 

This open up possibilities for a variety of applications, ranging 
from medical applications, such as home care and home medical 
treatment for aging societies; various types of design work 
applications, such as automobile design; educational applications, 
such as remote classes and remote experiments; and entertainment 
applications, such as games and recreation. An image of medical 
treatment is shown in Fig. 2. An automobile design coaction is shown 
in Fig. 3. 


Real-time object image recognition 
(1) Image recognition 

In order to display images of a highly realistic environment from 
the observer’s perspective, cameras are placed around the targeted 
natural and physical objects, and a method for switching the cameras 
position according to observer’s perspective is being considered. In 
actuality, this method does not offer sufficient realism because of the 
lack of continuity during image switching. 

To overcome these problems, the images of targeted natural and 
physical objects are first placed into a computer using Computer 
Vision, and then a method that creates from the observer’s perspective 
and displays images in real time is used. 



Fig.4 Real time recognition and generation of human image 
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This requires the technology to recognize targeted images, together 
with image creation and display technologies that create and display 
appropriate images, viewpoint and perspective detection technologies 
that detect the observer’s perspective, and other technologies. 

Fig. 4 provides details about the above technologies as they relate 
to human figures. Human figures are recognized and the wire frame 
model and texture are obtained and stared in the work station. Human 
figures are easy to model because they have many features in common, 
but natural and other physical objects are quite difficult to model 
because they come in all shapes and sizes. If a model can be created, 
the model is treated like a mannequin, and is then an easily 
recognizable target. However, new technologies will have to be 
developed in order to recognize natural and other physical objects that 
are difficult to model. 

(2) Recognizing movement information 

Information for head, hand and other movements by human figures 
will be detected in real time. Research is currently underway on non- 
contact movement detection methods as well as on detection methods 
in which sensors are placed on numerous parts of the human body. 

Movement information for natural and other physical objects, on 
the other hand, is not that easy to detect If we target a single tree, for 
example, some method will have to be found to detect the movement 
of each individual branch. 

(3) Creating images 

Information related to targets acquired in (1) as well as information 
related to movements acquired in (2) are used to create targeted 
images in real time using computer graphics. 

(4) Displaying images 

Images created in (3) are displayed on a large screen. 

Since shutter glasses are used to create a three-dimensional 
perspective, images corresponding to the left and right lenses are 
switched and displayed at high speed. A three-dimensional image is 
obtained by viewing the image with shutter glasses. 

A three-dimensional image will be achieved with the naked eye 
using a lenticular screen method that displays images corresponding re 
the left and right eye viewpoints, respectively, through 3.6-mm slits. 


Lenticular screen method is shown in Fig. 5. 

Coaction environments 

The next step will be to create a coaction environment where 
inhabitants will work together and take part in recreational activities, 
such as playing catch. In the coaction environment, object 
manipulation by words and gestures is used to play catch, to perform 
design work, and to control physical movement according to the laws 
of physics. If coaction environments interact, an entirely new coaction 
environment is created in line with the specific form of interaction. 

(1) Changes in the coaction environment due to interaction 

Once two environments A and B interact, a new environment that 
includes both the original environments is created. 

Coaction then occurs in the new environment. Fig.l shows the 
merging of two environments. 

(2) Basic manipulation, laws of physics, biological laws 

An object manipulation method based on words, gestures and other 
forms of communication will be used. Here the laws of physics, 
illustrated by objects falling, dishes breaking, physical changes from 
objects colliding, as well as sounds, are faithfully reproduced. 
Biological laws, such as plant growth and wilting in sunlight, are also 
faithfully reproduced. 

Future Prospects 

We have proposed a coaction environment in which people, 
animals, imaginary and other life forms work and play together in a 
highly realistic environment that includes real as well as unreal 
objects. These objects may be sent over communication lines from 
remote locations. 

By providing an environment that goes beyond reality as just 
described, people will be able to perform work and activities not 
possible in a real environment. This system will contribute 
tremendously to the welfare of humankind. 

An example might be providing all-night care for invalid elderly 
family members by artificial life forms that will notify the family, and 



Fig.5 Eye-Position Tracking Stereoscopic Display System 
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take appropriate action at the appropriate time when there is a 
problem. Such a system will also provide heretofore unfathomable 
benefits, like helping people to function in particularly difficult 
environments, such as underwater, underground and in outer space. 
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